{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f057401b-a4ee-462a-9ba9-3e342e06eb1a",
   "metadata": {},
   "source": [
    "## Lab 4 - Cross-encoder re-ranking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5a5536f0-651c-40e7-aa15-27ee0cda80b7",
   "metadata": {
    "height": 80
   },
   "outputs": [],
   "source": [
    "from helper_utils import load_chroma, word_wrap, project_embeddings\n",
    "from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3748b16d-d4a7-49c3-a48a-57dcfc42acd6",
   "metadata": {
    "height": 97
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "349"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_function = SentenceTransformerEmbeddingFunction()\n",
    "\n",
    "chroma_collection = load_chroma(filename='microsoft_annual_report_2022.pdf', collection_name='microsoft_annual_report_2022', embedding_function=embedding_function)\n",
    "chroma_collection.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68486bf8-37dd-4257-a23b-9ef50c47bcc5",
   "metadata": {},
   "source": [
    "# Re-ranking the long tail"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "888a86f8-2fe2-4682-bdaf-c15129ed1a32",
   "metadata": {
    "height": 165
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "• operating expenses increased $ 1. 5 billion or 14 % driven by\n",
      "investments in gaming, search and news advertising, and windows\n",
      "marketing. operating expenses research and development ( in millions,\n",
      "except percentages ) 2022 2021 percentage change research and\n",
      "development $ 24, 512 $ 20, 716 18 % as a percent of revenue 12 % 12 %\n",
      "0ppt research and development expenses include payroll, employee\n",
      "benefits, stock - based compensation expense, and other headcount -\n",
      "related expenses associated with product development. research and\n",
      "development expenses also include third - party development and\n",
      "programming costs, localization costs incurred to translate software\n",
      "for international markets, and the amortization of purchased software\n",
      "code and services content. research and development expenses increased\n",
      "$ 3. 8 billion or 18 % driven by investments in cloud engineering,\n",
      "gaming, and linkedin. sales and marketing\n",
      "\n",
      "competitive in local markets and enables us to continue to attract top\n",
      "talent from across the world. we plan to continue to make significant\n",
      "investments in a broad range of product research and development\n",
      "activities, and as appropriate we will coordinate our research and\n",
      "development across operating segments and leverage the results across\n",
      "the company. in addition to our main research and development\n",
      "operations, we also operate microsoft research. microsoft research is\n",
      "one of the world ’ s largest corporate research organizations and works\n",
      "in close collaboration with top universities around the world to\n",
      "advance the state - of - the - art in computer science and a broad\n",
      "range of other disciplines, providing us a unique perspective on future\n",
      "trends and contributing to our innovation.\n",
      "\n",
      "our success is based on our ability to create new and compelling\n",
      "products, services, and experiences for our users, to initiate and\n",
      "embrace disruptive technology trends, to enter new geographic and\n",
      "product markets, and to drive broad adoption of our products and\n",
      "services. we invest in a range of emerging technology trends and\n",
      "breakthroughs that we believe offer significant opportunities to\n",
      "deliver value to our customers and growth for the company. based on our\n",
      "assessment of key technology trends, we maintain our long - term\n",
      "commitment to research and development across a wide spectrum of\n",
      "technologies, tools, and platforms spanning digital work and life\n",
      "experiences, cloud computing, ai, devices, and operating systems. while\n",
      "our main product research and development facilities are located in\n",
      "redmond, washington, we also operate research and development\n",
      "facilities in other parts of the u. s. and around the world. this\n",
      "global approach helps us remain\n",
      "\n",
      "when the world around us does well. that ’ s what i believe will lead\n",
      "to widespread human progress and ultimately improve the lives of\n",
      "everyone. there is no more powerful input than digital technology to\n",
      "drive the world ’ s economic output. this is the core thesis for our\n",
      "being as a company, but it ’ s not enough. as we drive global economic\n",
      "growth, we must also commit to creating a more inclusive, equitable,\n",
      "sustainable, and trusted future. support inclusive economic growth we\n",
      "must ensure the growth we drive reaches every person, organization,\n",
      "community, and country. this starts with increasing access to digital\n",
      "skills. this year alone, more than 23 million people accessed digital\n",
      "skills training as part of our global skills initiative.\n",
      "\n",
      "also increased the number of identified partners in the black partner\n",
      "growth initiative and continue to invest in the partner community\n",
      "through the black channel partner alliance by supporting events focused\n",
      "on business growth, accelerators, and mentorship. progress does not\n",
      "undo the egregious injustices of the past or diminish those who\n",
      "continue to live with inequity. we are committed to leveraging our\n",
      "resources to help accelerate diversity and inclusion across our\n",
      "ecosystem and to hold ourselves accountable to accelerate change – for\n",
      "microsoft, and beyond. investing in digital skills the covid - 19\n",
      "pandemic led to record unemployment, disrupting livelihoods of people\n",
      "around the world. after helping over 30 million people in 249 countries\n",
      "and territories with our global skills initiative, we introduced a new\n",
      "initiative to support a more skills - based labor market, with greater\n",
      "flexibility and accessible learning paths to develop the right skills\n",
      "\n",
      "at times, we make select intellectual property broadly available at no\n",
      "or low cost to achieve a strategic objective, such as promoting\n",
      "industry standards, advancing interoperability, supporting societal and\n",
      "/ or environmental efforts, or attracting and enabling our external\n",
      "development community. our increasing engagement with open source\n",
      "software will also cause us to license our intellectual property rights\n",
      "broadly in certain situations. while it may be necessary in the future\n",
      "to seek or renew licenses relating to various aspects of our products,\n",
      "services, and business methods, we believe, based upon past experience\n",
      "and industry practice, such licenses generally can be obtained on\n",
      "commercially reasonable terms. we believe our continuing research and\n",
      "product development are not materially dependent on any single license\n",
      "or other agreement with a third party relating to the development of\n",
      "our products. investing in the future\n",
      "\n",
      "but generally include parts and labor over a period generally ranging\n",
      "from 90 days to three years. for software warranties, we estimate the\n",
      "costs to provide bug fixes, such as security patches, over the\n",
      "estimated life of the software. we regularly reevaluate our estimates\n",
      "to assess the adequacy of the recorded warranty liabilities and adjust\n",
      "the amounts as necessary. research and development research and\n",
      "development expenses include payroll, employee benefits, stock - based\n",
      "compensation expense, and other headcount - related expenses associated\n",
      "with product development. research and development expenses also\n",
      "include third - party development and programming costs, localization\n",
      "costs incurred to translate software for international markets, and the\n",
      "amortization of purchased software code and services content. such\n",
      "costs related to software development are included in research and\n",
      "development expense until the point that technological feasibility is\n",
      "reached, which for our\n",
      "\n",
      "fiscal year 2021 was a year of both successes and challenges. while we\n",
      "continued to make progress on several of our goals, with an overall\n",
      "reduction in our combined scope 1 and scope 2 emissions, our scope 3\n",
      "emissions increased, due in substantial part to significant global\n",
      "datacenter expansions and growth in xbox sales and usage as a result of\n",
      "the covid - 19 pandemic. despite these scope 3 increases, we will\n",
      "continue to build the foundations and do the work to deliver on our\n",
      "commitments, and help our customers and partners achieve theirs. we\n",
      "have learned the impact of our work will not all be felt immediately,\n",
      "and our experience highlights how progress won ’ t always be linear.\n",
      "while fiscal year 2021 presented us with some new learnings, we also\n",
      "made some great progress. a few examples that illuminate the diversity\n",
      "of our work include : • we purchased the removal of 1. 4 million\n",
      "metrics tons of carbon.\n",
      "\n",
      "we protect our intellectual property investments in a variety of ways.\n",
      "we work actively in the u. s. and internationally to ensure the\n",
      "enforcement of copyright, trademark, trade secret, and other\n",
      "protections that apply to our software and hardware products, services,\n",
      "business plans, and branding. we are a leader among technology\n",
      "companies in pursuing patents and currently have a portfolio of over\n",
      "69, 000 u. s. and international patents issued and over 19, 000 pending\n",
      "worldwide. while we employ much of our internally - developed\n",
      "intellectual property exclusively in our products and services, we also\n",
      "engage in outbound licensing of specific patented technologies that are\n",
      "incorporated into licensees ’ products. from time to time, we enter\n",
      "into broader cross - license agreements with other technology companies\n",
      "covering entire groups of patents. we may also purchase or license\n",
      "technology that we incorporate into our products and services.\n",
      "\n",
      "15 corporate social responsibility commitment to sustainability we work\n",
      "to ensure that technology is inclusive, trusted, and increases\n",
      "sustainability. we are accelerating progress toward a more sustainable\n",
      "future by reducing our environmental footprint, advancing research,\n",
      "helping our customers build sustainable solutions, and advocating for\n",
      "policies that benefit the environment. in january 2020, we announced a\n",
      "bold commitment and detailed plan to be carbon negative by 2030, and to\n",
      "remove from the environment by 2050 all the carbon we have emitted\n",
      "since our founding in 1975. this included a commitment to invest $ 1\n",
      "billion over four years in new technologies and innovative climate\n",
      "solutions. we built on this pledge by adding commitments to be water\n",
      "positive by 2030, zero waste by 2030, and to protect ecosystems by\n",
      "developing a planetary computer. we also help our suppliers and\n",
      "customers around the world use microsoft technology to reduce their own\n",
      "carbon footprint.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "query = \"What has been the investment in research and development?\"\n",
    "results = chroma_collection.query(query_texts=query, n_results=10, include=['documents', 'embeddings'])\n",
    "\n",
    "retrieved_documents = results['documents'][0]\n",
    "\n",
    "for document in results['documents'][0]:\n",
    "    print(word_wrap(document))\n",
    "    print('')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5665c695-22ea-4264-b1ac-5ba720b6d78b",
   "metadata": {
    "height": 63
   },
   "outputs": [],
   "source": [
    "from sentence_transformers import CrossEncoder\n",
    "cross_encoder = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c2a13d14-4484-46f0-8e67-277337f9d138",
   "metadata": {
    "height": 97
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores:\n",
      "0.98693466\n",
      "2.644579\n",
      "-0.26802942\n",
      "-10.73159\n",
      "-7.7066045\n",
      "-5.6469955\n",
      "-4.297035\n",
      "-10.933233\n",
      "-7.0384283\n",
      "-7.3246956\n"
     ]
    }
   ],
   "source": [
    "pairs = [[query, doc] for doc in retrieved_documents]\n",
    "scores = cross_encoder.predict(pairs)\n",
    "print(\"Scores:\")\n",
    "for score in scores:\n",
    "    print(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8ba6c8c5-9ce4-44d0-9223-6fdd77871f87",
   "metadata": {
    "height": 63
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New Ordering:\n",
      "2\n",
      "1\n",
      "3\n",
      "7\n",
      "6\n",
      "9\n",
      "10\n",
      "5\n",
      "4\n",
      "8\n"
     ]
    }
   ],
   "source": [
    "print(\"New Ordering:\")\n",
    "for o in np.argsort(scores)[::-1]:\n",
    "    print(o+1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f54cc00-eebc-4294-91bf-1a2cdce51708",
   "metadata": {},
   "source": [
    "# Re-ranking with Query Expansion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "377a84aa-1d93-4e97-9b2d-d59c46355338",
   "metadata": {
    "height": 165
   },
   "outputs": [],
   "source": [
    "original_query = \"What were the most important factors that contributed to increases in revenue?\"\n",
    "generated_queries = [\n",
    "    \"What were the major drivers of revenue growth?\",\n",
    "    \"Were there any new product launches that contributed to the increase in revenue?\",\n",
    "    \"Did any changes in pricing or promotions impact the revenue growth?\",\n",
    "    \"What were the key market trends that facilitated the increase in revenue?\",\n",
    "    \"Did any acquisitions or partnerships contribute to the revenue growth?\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ba0ed8ca-6640-4c09-9cb3-9de5e7cf46dc",
   "metadata": {
    "height": 97
   },
   "outputs": [],
   "source": [
    "queries = [original_query] + generated_queries\n",
    "\n",
    "results = chroma_collection.query(query_texts=queries, n_results=10, include=['documents', 'embeddings'])\n",
    "retrieved_documents = results['documents']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "28bac3a2-0d29-48dc-9b48-2d9313239a25",
   "metadata": {
    "height": 131
   },
   "outputs": [],
   "source": [
    "# Deduplicate the retrieved documents\n",
    "unique_documents = set()\n",
    "for documents in retrieved_documents:\n",
    "    for document in documents:\n",
    "        unique_documents.add(document)\n",
    "\n",
    "unique_documents = list(unique_documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "db9f2758-0f5a-49e5-b1fa-517b91324575",
   "metadata": {
    "height": 63
   },
   "outputs": [],
   "source": [
    "pairs = []\n",
    "for doc in unique_documents:\n",
    "    pairs.append([original_query, doc])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "aee59493-8a99-4da8-b94f-4747efcfc79d",
   "metadata": {
    "height": 46
   },
   "outputs": [],
   "source": [
    "scores = cross_encoder.predict(pairs)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f5eda9bc-ae76-4db6-9e0c-ae099d852d78",
   "metadata": {
    "height": 63
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scores:\n",
      "-10.042844\n",
      "-8.505108\n",
      "-10.00014\n",
      "-4.651892\n",
      "-9.80788\n",
      "-7.754099\n",
      "-9.918428\n",
      "-4.8184834\n",
      "-7.9171767\n",
      "-9.768024\n",
      "-1.136996\n",
      "-10.0839405\n",
      "-9.357721\n",
      "-11.0792675\n",
      "-6.90209\n",
      "-10.148884\n",
      "-4.3417687\n",
      "-5.141832\n",
      "-3.7948632\n",
      "-7.4906564\n",
      "-5.274752\n",
      "-3.7681527\n",
      "-10.711212\n"
     ]
    }
   ],
   "source": [
    "print(\"Scores:\")\n",
    "for score in scores:\n",
    "    print(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b1183e75-4c65-422e-bc47-48010d8b29c9",
   "metadata": {
    "height": 63
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New Ordering:\n",
      "10\n",
      "21\n",
      "18\n",
      "16\n",
      "3\n",
      "7\n",
      "17\n",
      "20\n",
      "14\n",
      "19\n",
      "5\n",
      "8\n",
      "1\n",
      "12\n",
      "9\n",
      "4\n",
      "6\n",
      "2\n",
      "0\n",
      "11\n",
      "15\n",
      "22\n",
      "13\n"
     ]
    }
   ],
   "source": [
    "print(\"New Ordering:\")\n",
    "for o in np.argsort(scores)[::-1]:\n",
    "    print(o)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fcd85cc-8898-41ed-a0aa-bd8a33fc565a",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c65337e9-85ee-47f7-89fd-7fe77cd0e1b2",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7794092-4195-4cf3-9eab-11c9c05a26b9",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2cab7a1-1be7-45f0-83b7-543e48f83901",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe0343be-73c9-4aed-83b0-aba09569ac87",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f0f3e33-e517-4f6b-8b38-c47c1e3d40b4",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a16fdcb1-57d0-4f04-af8f-7c7fc594d947",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "babe7893-9cbc-43c5-94ef-cbf8f5d68cf2",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60a9524b-1085-4bdf-a161-39f11397dc1f",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d189f088-b58e-4583-9590-afdfa624cf87",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b26a01a-4575-446b-b8dc-a8c5ab153172",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0950575b-b69d-46a3-8c91-c7af89f5c204",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f123ad8-b2e8-4a25-8b42-a520ecaf566b",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83c04587-d1de-419c-a213-2e3eb67dc33d",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3155972-824e-4ebe-a692-2227c113c5a8",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8144a4a-85f6-4800-87f9-36a1b6ceda1f",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ff0b18e-12a0-4ac0-97dd-8618b22e7dbf",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03ca7e7c-4b47-4652-9b46-a40b3dffa5e6",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f74e7d67-7f51-41c4-8e25-edbaa02d0bd8",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9188e886-d406-406f-b234-f5c3353a77a2",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d3bb286-2694-4ed4-8466-46865e997ced",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2876084b-4038-4b0c-8ec8-8294a86adfc1",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ac542e1-b094-431f-9611-cf7e36d3f0de",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcd6114b-c09d-4173-a623-9a08aaf63e4b",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad10ab65-b351-4f4b-b7d2-63474acfb9f9",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "800f3d81-cbdb-4ba4-8d49-85747fdfded8",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37847448-c9f6-4f51-bf06-f7809964a8b2",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2dcefc87-0964-4b94-946b-2145781ad606",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fc994bc-7b1e-476a-9df9-300a3e374882",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ef5f5d5-acb7-4b0a-93ef-e61306708e69",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44e4b33f-d8fb-4f3a-b884-8b43a3766583",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2a480a2-2c29-4a01-80dd-ee41934b7901",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8127c2bf-0d15-4b62-b46a-f7a17ad2ec92",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18ded129-a637-4269-a116-550fe9a90570",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1d7ee44-7b29-483f-a3f2-cc9d8e18880e",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e450dd8-9719-42c6-8c3c-33cac910e0a5",
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
